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(54) TiUe: 2-D MESH GEOMETRY AND MOTION VECTTOR COMPRESSION 



(57) Abstract 

An obicct of the invention is to provide a system and method for encoding and decoding the mesh and displacement of the node 
points from one fhime time instant to the next. This invention comprises methods for encoding the mesh data into a compressed fonnat and 
methods for decoding the compressed format The use of a compressed format facilitates efficient storage and conriraunicaiton of the mesh 
data. The coding mefhods described are lossless. i.c. the compressed format represents the same information contamed m the ongmal m^h 
data At the sa^ time, use of the compressed format reduces (he amount of storage space or communication tondwidth required The 
invention includes a system and method of encoding and decoding a dynamic mesh, including encodmg and decodmg a mesh geometry of 
a set of node points; and encoding and decoding a mesh node motion vector for each node point. 



i 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCX on the fiont pages of pamphlets publishing international applications under the PCX. 





Albania 


es 

FI 


AM 


Armenia 


AT 


Aostrta 


FR 


AV 


Australa 


GA 


AZ 


Azerbaijan 


GB 


DA 


Bosnia »nd Hcnxgovtna 


GE 


BB 


Barbados 


GU 


BE 


Bclgiuni 


GN 


BV 


Burkina Faso 


GR 


BG 


Bulgaria 


IIU 


BJ 


Deaia 


IE 


BR 


Braou) 


IL 


BY 


Bcbnis 


ts 


CA 


Canada 


IT 


CF 


Ccotial A frican Republic 


JP 


. CO 


Coago 


KE 


CH 


Switzcrtand 


KG 


CI 


COh: d'lvoirc 


KP 


CM 


Cantcmon 




CN 


China • 


KR 


CU 


Cuba 


KZ 


CZ 


Czech Republic 


LC 


DE 


Gcmumy 


U 


DK 


Dcnmait 


LK 


F.P. 


[Estonia 


LR 



SpaiB 

hwland 

France 

United Kingdom 

Georgia 

CBiana 

Guinea- 

Gieccc 

Iickutd 

bnei 

bxlaad 

lafy 

Japan 

Kenya 

Dcmocnuir Ptoplc's 
Rcpobtic of Kma 
Republic of Korea 
Kazirtroan 
Sakx Luria , 
Ltrcfatcnstctn 
Sri Lanka 
Ltbena 



IS 

VT 

Ul 

LV 

MC 

MD 

MG 

MK 

ML 

MN 

MR 

MW 

MX 

NE 

NL 

NO 

NZ 

FI. 

FT 

RO 

RU 

SD 

SF. 

SC 



Lesotho . 

Utbuania 

Uaxcmbourg 

l^via 

Monaco 

Republic of Moldova 

Madagascar 

The ftmner Yogoihif 

Rrpublic of Macedmia 

Mali 

Mntt$olia 

Mamitanb 

Mafcrvi 

Mexico 

Niger 

Ndheriands 

Mon»ay 

Nrw Zealand 

Poland 

Pcnugal 

Romania 

Russian Fcdcntion 

Sedan 

S«tden 

Singapore 



SI 


Slovenia 


SK 


Slovakia 


SN 


Senegal 


sz 


Swaziland 


TD 


Chad 


TG 


Togo 


TJ 


Ta^tuan 


TM 


Ttukmcnisfin 


TR 


Toriccy 


TT 


Trinidad and Tobago 


UA 


Ukraine 


UG 


Uganda 


US 


United Slatn nf America 


UZ 


Uibckisian 


VN 


Vict Nam 


YO 


Yugoslavia 


ZW 


Zimbabwe 



FCT/JP97/04607 



2-D MESH GEOMETRY AND MOTION VECTOR COMPRESSION 

Related Applications 
U.S. Patent Application Serial No. 08/447,418, filed May 23, 1995 for: "Video 

Compression System"; and 

U.S. Patent Application Serial No. 60/010,076, filed January 16, 1996 for 
"Video Coding and Decoding System". 

Field of the Tnvenrion 
This invention is in the field of coding and manipulation of images, and 
specifically, die coding of sequences of 2D meshes, generally corresponding to a 
sequence of images. 

Backrgmmid ^^^^ Tnvenrion 

A 2D triangular mesh refers to a tessellation of a 2D visual object plane 
into triangular patches. The vertices of the triangular patches are caUed "node points." 
The siraight-line segments joining the node points are caDed "edges."" 

A dynamic 2D mesh consists of a temporal sequence of 2D triangular 
meshes, where each mesh has the same topology (Le., structure), but node positions may 
differ from one mesh to the nexL Thus, a dynamic 2D mesh may be defined by the 
geometry of the initial 2D mesh and modon vectors at die node points for subsequent 
• meshes, where each motion vector points from a node point of die previous mesh in the 
sequence to a node point of the current mesh. The dynamic 2D mesh may be used to 

1 
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create 2D anixnauoos by mapping texture from a still unage onto successive 2D meshes 
via weU-known texnne mapping methods. For example, the dynamic mesh may be used 
torenderawavingnagfromastiUimageofaflag. TT.e local defotmauons of th4 
textuie in time are captured by the motion of mesh nodes ftom one mesh to the next 
Hence, different animations of the same texmre may be achieved by different sets of 

node modon vectors. 

Texmre mapping utilizes the strucmre of the mesh. Le., the way the nodes 
of the mesh are connected with each other, namely the configuration of the edges of the 
mesh. AmeshmayhaveaspedfiedimpUciLstrucmre.snchasmufonhstmcmreor 
Delaunay strucmre. as described in S. M. Omohmidro, "The Delaunay triangulation and 
function learning." Intemadonal Computer Science Instimte Technical Report TR-90- 
001. University of California Berkeley. January 1990. 

Efficient coding of an animation sequence may be achieved by separately 
coding the still image texmre. and ±e associated 2D mesh. i.e., die geometry and node 
vectors The associated 2D mesh is represented by die geometry of die first mesh and 
niouon veaors of die nodes of dns first and subsequem meshes. The 2D mesh is 
encoded by coding dxe geometry of die first mesh and modon vectors of die nodes of 

this first and subsequent meshes. 

The mesh geometry compression technique described here is limited to 2D 

) triangularmesheswidiimplicittopology.specificaUymeshesw^^ 

topology. In diese cases, die mesh topology is defined impliciUy. given die locadons of 

die mesh nodes (also called vertices) and some addidooal information to be specified in 

detail later. Algorimms to implement Delamiay trianguladons are available in literature 
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and are doi described here. It should be aoted that Delaunay trianguladons are uniquely 
defined except if the nodes to be triangulated contain certain degeneracies in their 
locations. Here, it is assumed that both the mesh encoder and decoder use an agreed 
upon technique to handle such degeneracies. Such techniques are well known to those of 
5 qViTI in the art. The mesh geometry compression technique described here allows a high 
compression ratio for these constrained classes of meshes. 

Representing mesh motion efficiently is important for describing mesh- 
based animations. Here, we describe a technique for compression of mesh motion in the 
2D case, although it should be noted that the principle may be extended to the case of 3D 

10 meshes with 3D motion straightforwardly. Furthermore, it should be noted that the mesh 
motion compression technique described here is directly applicable to meshes with 
general topology, although the examples provided herein describe meshes with 
consumed topology. Finally, it should be noted that the principles of the invention with 
respect to motion coding may also be applied to the coding of surface appearance 

15 attributes snaightforwardly. 

The coding methods described here may for instance be employed in the 
context of MPEG-4. MPEG-4 is an object-based multimedia compression standard being 
developed by the Motion Picture Experts Group, which allows for encoding of different 
audio-visual objects (AVO) in the scene separately, as an extension of the previous 

20 MPEG- 1/2 standaitls. These AVO aie decoded and then composited at the user terminal 
according to a transmitted scene description script and/or user interaction to form display . 
frames. The visual objects may have namral or synthetic content, including audio, video, 
3D graphics models, scrolling text and graphics overlay, and so on. 

3 
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f^^ Tpmarv of f^'^ Tnveniion 
This invention comprises methods for encoding iht mesh data into a 
compressed format and methods for decoding die compressed format The use of a ^ 
5 compressedformatfaciUtatesefficientstorageandcommunicationofthemeshda^ Tne 

coding methods described are lossless. Le. the compressed format represents the same 
. information contained in ±e original mesh dam. At the same dme, use of the 

compressed format reduces the amount of storage space or communication bandwidth 
reqnired. 

10 The inventions includes system method of encoding and decoding a 

dynamic mesh, including encoding and decoding a mesh geometry of a set of node 
points; and encoding and decoding a mesh node modon vector for each node point. 

An objea of the invention is to provide a system and method for encoding 
and decoding the mesh and displacement of the node points from one frame time instant 

15 to the next 

Pn>.f nff^rrijin^n r)f r^'r r)ra wings 
Rg- 1 depicts a mesh-based motion modeling using a triangular mesh. 
Rg. 2 depicts node motion and deformation of an image object when a 
20 mesh is imposed thereover. 

Fig. 3 depicts a typical encoder/decoder system. 

Hg. 4 depicts an animaiion application being processes by the system of 



Fig. 3. 



25 



Hg. 5 depicts a generalized uniform 2D mesh. 
Fig. 6 depicts foor types of uniform meshes. 
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Fig. 7 depicts a nodfe point ordering in a 2D triangular mesh of the 

Delaunay type. 

Fig. 8 depicts node point ordering in a 2D triangular mesh of the uniform 

type. 

Fig. 9 depicts a halfway-point in a breadth-first traversal of a 2D 

triangular mesh- 
Fig. 10 depicts a breadth-first traversal of triangles in a 2D mesh. 
Hg. 1 1 depicts a set of decoded node points and mesh boundary edge 

segments. 

Fig. 12 depicts a decoded triangular mesh obtained by constrained 

Delaunay iriangulation. 

p^ta ?^^^ Description of r hp. Preferred Embodiment 
Mesh-Based Motion Modeling for Video Processing 

Modeling and estimating the motion of objects in a sequence of image 
frames is common in video processing and has a variety of appUcadons. A common 
approach is block-based motion modeling, where motion parameters are estimated for 
each square block of pixels independendy. Specifically, the oanslational block motion 
model has proven satisfactory for video compression; and has been adopted in 
international standards such as MPEG-1, -2 and ^. Recently, 2D mesh-based motion 
modeling has been proposed as a promising alternative in video processing to block- 
based motion modefing. 

A 2D mesh is a tessellation (or partition) of a 2D planar region into 
polygonal patches. In the foUowing, the vertices of the polygonal patches are referred to 
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as the node points of the mesh. IHe patches are usuaUy triangles or quadrangles, leading 
to triangular or quadrilateral meshes, respectively. See Fig. la. In 2D mesh-based 
motion modelmg. a 2D mesh is associated with an image or a set of images, such that the 
polygonal mesh pa«:hes correspond to patches of texmre in the images, where the texture 
consists of the image pixels inside a patch. Polygonal patches in a reference frame are 
deformed by the movements of the node points into polygonal patches in another frame, 
and the texture inside each patch in the reference frame is warped onto.the other frame 
using a parametric mapping as a funcdon of the node point motion vectors. See Figs. lb 
and ic. For triangular meshes, affine mapping i^used. which may model translation, 
rotation, scaling and shear. Note that, in mesh-based motion modeling, the patches do 
uot overlap in the reference frame or in d.e current frame. As used herein, a sutement 
that two triangles are adjacent means that they share a common edge. 

Assuming that proper consults are imposed in the parameter estimation, 
an affine transform can guarantee the continuity of the mapping across die bomidaries of 
adjacent triangles- This implies that the original 2D motion field may be compactiy 
represented by the motion of ±e node points, from which a continuous, piece-wise affine 

motion field can be reconstructed. 

3D polygon meshes have long been used for efficient 3D objea geometry 
modeling and rendering in computer graphics. Equations similar to parametric mappings 
used in mesh-based motion modeling have also been used in 3D graphics to perform 

texmre mapping, a popular procedure to render namral images on polygon meshes 
describing graphic objects for phott^realistic synthesized images. Texmre mapping in 
3D graphics is realized by assigning a texmre coordinate (a pixel position on a 2D image) 
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to every 3D node point on the polygonal mesh. Thus, each polygonal surface element on 
the 3D mesh is associated with a patch of the 2D image, which is then rendered on the 
polygon mesh subject to proper warping transformation. An animation may be created 
by rendering the same image onto a defomiing mesh repeatedly. A similar process may 

5 be performed with 2D meshes to tender an ammaied image sequence from an image and 
corresponding deforming meshes. 
Motion Estimation for Mesh-Based Video Processing 

Determining the motion of 2D meshes from a given image sequence is 
performed by estimating the motion of mesh node points over time. In the case of 2D 

10 mesh-based motion modeling, motion estimation refers to searching in a given reference 
image for the best locations of the node points, such that the triangular image patches in 
the reference frame optimally match those in die current image. The mesh in the initial 
reference image may have a regular structure, in which case it is called a uniform mesh, 
or it may be adapted to the image, in which case it is called a content-based mesh. A 

15 description of an algorithm for content-based (adaptive) mesh design may be fomid in the 
literature. 

Various techniques have been proposed for node motion vector search . 
from one frame to the next. THe simplest method is to form blocks that are centered 
around the node points and dien use a gradient-based technique or block-matching to find 
20 motion vector at the location of the nodes. Hexagonal matching and closed-foim 

matching techniques find the optimal motion vector at each node mider the parametric 
waiping of all patches surrounding the node whHe enforcing mesh connectivity 
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consoaints. Another method is iterative gradient-based optaxnizadon of node pomt 
locations, taking into account ixnage feamres and mesh defonnation criteria 

Referring now to Fig. 2, an image object 10 is depicted. The motion of 
nodes, such as nodes 12. 14. &om an image, object 10 is described as the ixnage object is 
transformed into an animated image object 16. having nodes 12a. 14a. TTie motion of 
nodes describe local motion and deformation of image object 10 over which a mesh 1 8. is 
imposed. The transformation that takes the mesh of object 16 to image object lOresults 
in local motion and deformation as weU as a scaling of the entire image object 

As previously noted an object of this invention is to provide a method for 
encoding and decoding the mesh and displacement of the node points from one frame 
time instant to the next. The texn^ to be animated may be encoded using a still image 
coding method, such as those used in MPEG-4 (Moving Picmre Experts Group Standard 
4). THe texmre to be animated may be a stiU image of a namral object or it may be a 

syntheuc (computer generated) image. 

The appHcaiipn at the decoder decodes die texture and the mesh and 
renders the animation. Tb. current version of MPEG-4 Systems Working Draft (WD) 
specify a framewodc for applic^ons that support animation. MPEG-4 Systems Working 
Draft V 5.0 of ISO/IEC 14496. Docmnent No. N1825. July 1997, hereinafter MPEG-4 

WD V 5.0). 

2D Mesh Encodo- 

Initially, the encoding of the mesh geometry and then encoding of mesh 
morion vectors will be described. Assmne that the mesh structure (i.e.. topology) , is 
known and is either a umfonn mesh or a Delaunay mesh. 

8 



* - 

■5 . ' 



WO 98/27515 




PCT/JP97/04607 



Referring to Figs. 3 and 4, a typical system that may use the method of the 
invention is depicted, and includes an encoder subsystem. 20. shown in Fig. 3. 
Subsystem 20 includes a texture encoder 22, which receives still image texmre data, 24, 
and generates a texture coded bit stream, 26. A mesh encoder 28 receives a 2D mesh 

5 sequence 30. Encoder 28 includes a geometry encoder 32 and a motion vector encoder 
34. Encoder 28 generates a mesh coded bit stream 36. 

The decoding subsystem 20 is shown in Fig. 4, and includes a texture 
decoder 38 and a inesh decoder 40. Mesh decoder 40 includes a geometry decoder 42 
and a motion- vector decoder 44. Output from texture decoder 38 and mesh decoder 40 

10 results in texture mapping data 46, which.is used in an animation application 48. 

resulting in an animated image 50. It will be appreciated by those of skill in the an that 
while the preferred embodiment of the system of the invention is described as 
manipulating a still image, other images, such as video images, may also be manipulated. 
■fjff^h r^nme-mi Encoding 

15 Because the initial 2D triangular mesh is either a uniform mesh or a 

DeUunay mesh, the mesh triangular topology (links between node points) is not encoded; 
only the 2D node point coordinates p. = (x., yj are encoded. In the bit stream, a special 
flag may specify whether the initial mesh is uniform or Delaunay. See Table 8, below. 

In case of a uniform mesh, five parameters, iir_of_niesh_nodes_hori, 

20 nr_or_mesh_nodes_yerti. ined»_rect_size_hori, inesh_rech_size_yerti, and 
triangle_split_code, (further defined and described in Table 5. below) are used to 
specify the complete topology and node point locations. An example of a 2D uniform 
mesh is shown in Fig. 5, generally at 60. wherein die number of nodes 62, shown as dark 

9 
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Circles for emphasis, in the horizontal and vertical dimensions are equal to 5 and 4, 
respectively. The splitting of rectangles into triangles may be done in four different 
schemes, as shown in Fig. 6. One of these types is shown in Fig. 5. which corresponds to 
triangl€_spBt_code = 'OO'. Other codes are further defined in Table 9. The first two 

5 parameters. . nr_of_mesh_nodes_hori, nr_or_inesh_nodes_verti. specify the number 
of aodes in the horizontal and vertical direction, respectively, of the unif onn mesh. The 
next two parameters, mesh_rect_si2e_hori. mesh.rect.size.verti. specify the 
horizontal, and vertical size of each rectangle (containing two triangles) in half pixel 
units, respectively. This specifies the layout and dimensions of the mesh. The last 

10 parameter. triangle_spUt_code, specifies how each rectangle is spUt to form nvo 

triangles. 
T ;)clanTiav mesh 

The node point coordinates are encoded, by first encoding the boundary 
node points and then the interior node points of the mesh. To encode the interior node 

15 positions, the nodes are traversed one by one using a nearest neighbor strategy and each 
node position is encoded differentiaUy using the position of the previously encoded node 
as a predictor. A linear ordering of the node points is computed such that each node is 
visited once. When a node is visited, its position is differentiaUy encoded with respect to 
the previously encoded node. Thai is. the difference between the position of the present 

20 node and the reconstructed value of the previous node is encoded using variable length 
coding (VLC). The ordering is such thai the boundary nodes are visited first: then the 
interior nodes. By sending the total number of node points and the number of boundary 
node points, the decoder knows how many node points wiD foUow, and how many of 

10 
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those are boundary nodes; thus it is able to reconstnict the polygonal boundary and the 

locations of all nodes. 

This procedure is Ulustrated in Fig. 7, which is m example of a small 
triangular mesh. 70. Fig. 7 depicts a traversal of node points of a 2D triangular mesh and 
5 ordering of the node points to be coded. First, the boundary nodes, Po-P,, are visited 
according to connectivity, Le.. the next node is always the next connected node on the 
boundary in the counterclockwise direction. Then, the interior nodes, p,o .-P,4. are visited 
according to proximity. i.e., the next node is always the nearest node that is not already 
encoded. The total number of nodes and the number of boundary nodes is encoded. The 

10 top-left node po is encoded without prediction, where the top left node po = (x,, %) is 
defined as the node n with minimum x. +y.. assuming the origin of the local coordinate 
system is at the top left If there is more than one point with the same value of + y. , 
then choose the node point with minimum y. Then, the next counterclockwise boundary 
node pi is found and the difference between Po and p, is encoded; then all other 

15 boundary nodes are encoded in a similar fashion. Then, the not previously encoded 

interior node that is nearest to the last boundary node is found and the difference between 
these is encoded. That node is defined as the not already encoded node n with minimum 
K - ^1 +|y« - >i»l ' ^^^^ '^^ ^ represent the coordinates of ±e previously 
encoded node. Then, the not previously encoded node nearest to the last encoded node- is 

20 found and the difference is encoded, and so on. Every node, pobi has an x- and y- 

coordinate, p, = {\. y.), each of which is subtracted from the corresponding coordinate 
of the previously encoded node point The two resulting difference values are encoded 
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using variable length coding (VLC). In a particular embodiment of the currem invention, 
variable length codes specified by MPEG-4 for coding sprite trajectories are used. In 
principle, specific variable length codes may be designed. 
T?T^^fy1in|y of |^^T^ff Mprinn Vectors 
5 Each node point p, of a 2D mesh numbered A in the sequence of meshes 

has a 2D motion vector v. . defined from mesh k to M. The locations of node points p.' 

of die mesh numbered are encoded by encoding all motion vectors v. . n = 0 N - 

1. Note that the triangular topology of the mesh remains die same throughout the 
sequence. 

^0 Two alternative methods for node morion vector coding will be described, 

both of which incorporate predictive coding of die node motion vectors. Predictive 
coding of motion vectors entails prediction of each motion vector by one or more already 
encoded motion vector(s) of odier node points in die mesh; subseqnendy the d^erence 
between die predicted vector and acmal motion vector is encoded instead of the original 

15 The first mediod, Mediod L uses only one already encoded motion veaor to predict the 
value of a particular motion vector. The second mediod. Method H, uses two already 
encoded motion vectors to pretfia the value of a particular motion vector. TTie methods 
also differ widi respect to the choice of predicting motion vectors; in Method I the 
predicting motion vector is defined as die preceding motion vector in a simple node point 

20 orderinr-inMethodnthepiedictingmotionvectorsaredefinedbydieuseofabreadtii- 

first traversal of the mesh. Note that the bit stream syntax for mesh motion coding, as 
defined in Table 6. is the same for Method I aiid n. 
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Assume there is a unique ordering of ±e available node points p„ . where 
n denotes the ordering number. In the case where the mesh is a Delaunay-type mesh, as 
shown in Fig. 7. the ordering is simply defined as the order in which the node point 
5 locations are encoded during mesh geometry encoding; i-e.. the node point for which the 
location is encoded first is . the node point for which the location is encoded after that 
is p,, etc. The ordering used during mesh geometry encoding is based on a traversal of 
the initial mesh, where boundary nodes are visited first, then interior nodes are visited, as 
previously described. 

10 In the case where the mesh is a uniform-type mesh, as shown in Fig. 8, the 

node ordering is defined on the basis of the mesh structure as follows. Tne first node 
point in die ordering. S,. is the most top-lefi node point of the uniform mesh; die 
foUowing node points in the ordering are obtained by traversing the node,points of the 
uniform mesh from left to right in the first (top) row of node points; going to the node in 

15 the second row of nodes immediately below the last node of the first row; traversing die 
second row from right to left; going down to the third row of nodes and traversing the 
third row from left to right, etc: This ordering is defined by a simple traversal of the 
uniform mesh. 

For each node point in the mesh, a one bit flag, the 
20 iiode_motion_vector_flag. is encoded to specify whether diat node point has a non-zero 
motion vector. For each node point in die mesh widi non-zero motion vector, a modon 
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veaor difference .ector e„ is encoded to specify the difference between the motion 
veoor of that node and the previously encoded motion veadr. 

Thus, the encoding process is as foUows: The motion vectors of aU node 
points p.. n = 0, iV - 1. are defined by: 

If rhe first node point in the ordering defined above has a zero motion vector. i.e. 
7o = (0.0). no morion vector data is encoded for this node point If the first node point 
has a non-zero motion vector, then a vector e, is defined, equal to the motion vector of 
the first node point: 

- - (2) 
Co = vo . 

The vector ^ is encoded by VLC of its x- and y-componenL For aD other node points in 
the ordering as defined above, die encoding process is as foUows: 

If node point p. . n = 1. .... iV- 1. has a zero motion vector. Le. v. = (0,0), 
then no morion vector data is encoded for that node point. Otherwise, a morion vector 
difference vector e" is computed from die previously encoded motion vector and die 
cturent motion vector by: 

■ i, = v.-v^.. . 
Then, die vector e. is encoded by variable lengdi encoding of its x- and y-componenL 
Thus, to encode die acmd motion vectors, difference vectors 5. are encoded one by one 
in the order specified above. 
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Method n: M (?ri»n Vfictnr Predicdon 

Method n is based oo a predictive coding technique where two motion 
veaors are used to compute a predictor. The predictive coding technique of Method H 
specificaUy employs the foUowing technique of motion vector prediction. To encode the. 

5 motion vector of a node point p, diat is pan of a triangle t^ = <p,. p„. p.). where the two 
motion vectors vectors v, and v„ of the nodes p, and p„ have already been encoded, one 
may use the values of v, and v„ to predict v^, and encode the prediction error value. 
Starting from an initial triangle t^ , for which all three node motion vectors have been 
encoded, there must be at least one odier. neighboring, triangle t^ that has two nodes in 

10 common with t,.. Because the motion vectors of the two nodes that t^ and t. have in 
common have already been encoded, one may use these two motion vectors to predict the 
motion vector of the third node in t^. The acmal prediction vector w, is computed by 
averaging of the two prediction motion vectors and the components of the prediction 
vector are rounded to half-pixel accuracy, as follows: 

15 w, = 0.5 X (|_v„ + V, + 0.5j). (4) 

This prediction formula is used for all motion vectors, except for the first and second 
motion vectors encoded. When encoding the first motion vector, the prediction vector is 
set to the zero vector 

w^=(0,0). (5) 
20 When encoding the second motion vector, the prediction vector is set to the first encoded 
motion vector 



w = V 

oi ft 



(6) 
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For each node pointn. .predicnon error veaor. e" . ^ computed by taking the difference 

between the prediction vector and the actual motion vector 

- - - (7) 
e. =v„-w. . 

Each prediction error vector is encoded using variable length eacoding. This procedure 
5 is repeated while traversing the triangles and nodes of the mesh, as explained below. 
Note that the prediction error vector is encoded only for node points with a non-zero 
motion vector. For all other node points, die motion vector is simply v. = (O.O)- The 
previously identified node_inotion_vector_flag is used to specify whe Aer a node has a 
non-zero motion vector or not. 

10 MflT tl"^ TT: M rfrh Traversal 

A breadth-first traversal is used to visit all the triangles and nodes in the 

mesh numbered it, and to encode the motion vectors defined from mesh * to The 
breadth-firsr traversal is a uniquely defined traversal of the mesh such that every triangle 
is visited exactly once and each node is visited ai least once. Because this traversal is 
15 determined by the topology of the mesh, which is known at the encoder at aU times and 
- known ai the decoder once die iniual mesh has been decoded, die traversal of d.e mesh 
may be performed at both die endoder and decoder in exactiy die same manner. 

The breaddi-Grst traversal of die mesh triangles is defined as follows and 
is illustrated in Hg. 9. generally at 90. First, define an initial triangle 92 as follows: 
20 Define die top left mesh node 94 as die node n widi minimum x„ + y. . assmning die 
origin of die locaT coordinate system is at die top left. If diere is mor^ dian one node 
widi die same value of v Yo ^ ^len choose die node point among diese widi minimum y 
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The edge 96 between the top-left node of the mesh and the next clockwise node 98 on the 
boundary is an edge of the triangle that is designated.as the initial triangle. Label the 

initial triangle with die number 0. 

Second, all other triangles are iteratively labeled with numbers 

5 1 . 2, .... M - 1 . where Af is the number of triangles in the mesh. Among all labeled 

triangles that have adjacent triangles which are not yet labeled, find the triangle with the 
lowest nmnber label This triangle is referred to in the following as the current triangle 
100. In the first iteration, the imriaZ triangle becomes die currenr triangle. Define die 
base edge of diis triangle as the edge that connects this triangle to die already labeled 

10 neighboring triangle with the lowest number. In die case of die initial triangle, die base 
edge is defined as die edge between die top-left node and die next clockwise node on die 
boundary. Define die right edge of die. current triangle as die next counterclockwise 
edge widi respect to die base edge; and define die left edge as die next clockwise edge 
widi respect to die base edge. That is. for a triangle t, = (p„ p.. p.). where die vertices 

15 are in clockwise order, if (p,pj is diebaseedge. dien (p,pj is die right edge and 

(p,pj is die left edge. 

Third, detennine if ttiere is an unlabeled triangle adjacent to die current 
triangle, sharing die right edge. If diere is such a triangle, it is labeled with die next 
available number. Determine if diere is an unlabeled triangle adjacent to die current 
20 triangle, sharing die left edge. If diere is such a triangle, it is labeled widi die next 
available number. This process is coiitinued iteratively until aU niangles have been 
labeled widi a unique number m. Referriing now to Fig. 9, a breaddi-first traversal of a 
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2D triangular example mesh is depicted. Hie traversal is halfway through die mesh - 
five triangles have been labeled (widi numbers) and the motion'veaors of six node 
points have been encoded (marked with a box symbol). The triangle which has been 
labeled 'S' is the -current triangle' : the base edge is 'b'; the right and left edge axe V and 
•r . The triangles diat will be labeled next are the triangles sharing the right, resp. left 
edge with the current triangle. After those triangles are labeled, the triangle which has 
been labeled *4'. will be the next 'cunent triangle* and another motion vector will be 
encoded 

The ordering of the triangles according to their assigned label numbers 
impHcidy defines the order in which the motion vector data of each node point is 
encoded, initially, motion vector data for die top-left node of the mesh is encoded. No 
prediction is used for die motion vector of diis node, hence this data specifies the motion 
vector itseli men, motion vector data for die second node, which is the next clockwise 
node on die bomidary with respect to die top-left node, is encoded. This data contains 
the prediction error for die motion vector of diis node, where die motion vector of die 
top-left node is used as a prediction. THese first two nodes diat form die base edge of die 
initial ffiangle are madced with die label 'done'. 

During each iteration of die breadth-first traversal, as described above, 
and starting &om die inidal triangle, die motion vectors of die two nodes diat are on die 
base edge of die current triangle during dial iteration are used to form a prediction for die 
motion vector of die diird node of diat triangle. If diat diird node is not yet labeled 
'done', prediction error values are computed by subtracting die prediction from die actual 
morion vector, and die motion vector data is encoded by VLC. -me diird node is labeled 
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•done'. If the thiid note is already labeled 'done', it is simply ignored and no data is 
encoded. Note that due to the namre of the traversal process, the two nodes on the base 
edge of a triangle are guaranteed to be labeled 'done' when that triangle becomes the 
'current triangle', signifying that their motion vectors have already been encoded and 
may be used as predictors. Motion veaors and triangle queue states for the encoding 
steps of the method are shown in Table 1. 



10 



encode 


•triangle queue state 


V3. 


{ta} 




{ts.. to} 


V7 


{to. ta} 


Vo 


{t8. ta} 


Vb 


{ta. t9. U] 


V4 


{t9.t6.ti} '■ 


V9 


{ts. ti} 




{t,. t7} 


V2 


{tr. U) 


Vb 


{U} 




{} 


Table 1 



The breadth-first traversal of the triangles and the encoding of the node 
motion vector data may be performed simultaneously by making use of a Crst-in-firsi-out 
(FIFO) queue of triangles. It should be noted, however, that the following description 
15 serves as a preferred embodiment of the invention, and that implemenution of the 
modon vector encoding process using a triangle queue is only one of several ways to 
implement the breadth-first traversaL 
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The FIFO queue is used to temporarily store triangles t^, where each 
triangle is specified by an ordered triple of node points t, = (p,. p.. p.). such that the. 
node points of the triangle are stored in clockwise order and such that the first and second 
node points always correspond to the base edge of a triangle. 
5 The top-left node of the mesh and its neighboring node on the boundary, 

in a clockwise direction, together define the initial triangle in the traversal. The motion 
vectors associated with these two nodes are encoded as described above. The (initially 
empty) FIFO queue is now initialized by appending the initial triangle at the end. 
Furthermore, the initial triangle is labeled ^.:m number 0.- and its two already-processed 
10 nodes are marked 'done'. Next, the FIFO queue is processed as follows mitil it is empty. 

The ordered triple = (p,. p.. P.) that is at the head of the queue is 
removed from the queue. The base edge, right edge and left edge of this triangle may be 
identified immediately from this triple. If the third node. p.. is marked 'done', no 
further action is taken; otherwise, a prediction vector is computed using the already 
15 encoded motion vectors associated with ±e first two nodes in the triple, p, and p„. and 
the acmal motion vector at p.. The prediction error values are encoded by VLC. The 
subject node point now mariced 'done'. Determine whether t, has an adjacent triangle 
sharing the right edge that has notyet beenlabeled. If there is such a triangle, it is 
labeled with anew nmnber and appended to the end of the queue. Determine whedier t, 
20 hasanadjacentuianglesharingtheleftedgethathasnotyetbeenlabeled. Ifthereis 
such a triangle, it is labeled with a new number and appended to the end of the queue. 
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When all triangles have been labeled with a numbfer, no more triangles 
will be appended to the queue. When the queue is empty, all triarigles will have been 
processed and all node motion vectors will have been encoded. 

Because every triangle in the mesh has at least one adjacent neighbor, and 
5 triangles aie labeled when visited, every triangle is visited exactly once and the traversal 
terminates when (and only when) all triangles have been visited. The breadth-first 
traversal of triangles defines an ordering in which node points are visited. Node points 
may be visited more than once, but their motion vectors are encoded only at the first time 
a node is visited. Each node is labeled at the time of encoding, such that no anempt will 

10 be made to encode the corresponding motion vector again. The unique ordering of node 
points corresponds to the order of motion vector data put in the bit stream. 

The breadth-first traversal process is illustrated in Fig. 10 for a small 
triangular mesh, showing a traversal of triangles arbitrarily numbered to , -., i, and the 
corresponding ordering of node points arbitrarily numbered p^, .... p,. The initial 

15 triangle tj is defmed using the top-left node and the next cloclcwise node on the 

boundary p,. The mbdon vector v, is encoded without using any prediction; the motion 
vector Vi is encoded using only v, as a prediction. Nodes and Pi are marked 'done*. 
The triangle queue is initialized witii t, and t, is labeled. The acmal breadth-first 
traversal of the mesh now starts by removing the front element from die queue, in this 

20 case tj . The motion vector of its third node. % . is now encoded using % and v^ as 

predictors and putting the prediction oror in die bit snream. Node p^ is marked 'done'. 
The next step is to append any triangles adjacent to the cuneni triangle tj and not yet 
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labeled to che triangle queue. The triangle on the right of the current triangle (in this case 
is) is appended first; then die triangle on the left (in this case t^); both are labeled. 

The next iteration of the traversal commences by removing the front 
element from the queue, in this case . The modon vector of its third node, , is now 
encoded using and v^j as predictors and retrieving the prediction error from the bit 
stream. Node py is marked 'done' and the connected triangle t, is appended to the 

triangle queue and labeled. 

Such iterations continue until the triangle queue is empty and all node 
motion vectors have been encoded. The steps of the algorithm are illustrated in Table 1. 
2D Mesh Decoder 

In reference to the MPEG-4 syntax, described later herein, and again 
referring to Fig. 4 and Tables 3-6, after inesh_object_start_code has been decoded by 
mesh decoder 40, a sequence of mesh objea planes is decoded, until a 
mesh_object_end_code is detected. Thie new_mesh_flag of the mesh object plane class 
determines whether the data that follows specifies the initial geometry of a new dynamic 
mesh, in which case the data is sent to geometry decoder 42, or whether it specifies the 
modon of nodes from the previous mesh to the current mesh, in a sequence of meshes, in 
which case the data is sent to motion vector decoder 44. 

In the following, the decoding of mesh geometry is described, followed by 
a description of the decoding of mesh motion. This description is general in nature and 
may be implemented using any syntax other than the MPEG-4 syntax specified herein in 
the preferred embodiment. 
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Mftsh Geom fifTV Hftcnding 

Because the initial 2D triangular tfiesb is either a uniform mesh or a 
Delaunay mesh, the mesh triangular topology (links between node points) is not encoded; 
only the 2D node point coordinates p„ = (x.. yj are encoded. The niesh_type_code 
(Table 8) specifies whether the initial mesh is uniform or Delaunay. 
TTnifnrm mesh 

As previously stated, five parameters specify the geometry of a uniform 
mesh CTable 5). The JBrst two decoded parameters specify the number of nodes in the 
horizontal and vertical, respectively, direction of the uniform mesh. The next two 
decoded parameters specify horizontal and vertical size of each rectangle (containing two 
triangles) in units accurate to half pixel units. The last parameter specifies how each 
rectangle is split into two triangles. 
DRlarmav mesh 

First, the total nimiber of node points in the mesh N is decoded; then, the 
number of node points diat are on the boundary of the mesh is decoded. Note that N 
is the sum of the number of nodes in the interior of the mesh. N, and the number of nodes 
on the boundary, A/», 

N = M. + N,. ' (8) 

Then, the locations of boundary and interior node points are decoded- The origin of the 
local coordinate system is presumed to be at die top left of the image object 

The X-, and y-coordinates of the first node point are decoded, which is 
specified in half-pixel units by a fixed length code (FLC). All die other node point 
locations are computed by adding differential values to previously decoded node 
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locaiions. In particular, a delta_x, and delta_y value is added to the x- and y- 
coordinate. respectively, of the previously decoded node location. The delta.x and 
delta_y value are each decoded by first decoding a variable length code (VLE) 
specifying their respective lengtfis. de!ta_xjen_vlc and delta_y_len_vlc, and then 
5 decoding their values. Thus, die coordinates of die initial node point Po = % ) is 
decoded as is. The coordinates of all odier node points, p. = (x,.yj are determined by 
adding a decoded value to the previously decoded node point coordinates: 
X =x , + <ir, and y„ = y^i + ify, . The differential values are decoded using variable 

length codes used at die encoder. 

IQ The ordering in the sequence of decoded locations is such that the first 

locations correspond to boundary nodes (in counterclockwise direction). Thus, after 
receiving the first N, locations, die decoder is able to reconstruct die boundary of die 
mesh by connecting each pair of successive boundary nodes, as wefl as the first and the 
last, by straight-line edge segments. The next - N, values in die sequence of decoded 

15 locaiions correspond to interior node points. Hius. after receiving // nodes, die locations 
of bodi die boundary and interior nodes may be reconstructed, in addition to die 
polygonal shape of die boundary. This is illustrated in Rg. 11, which depicts decoded 
node points (rectangles) and mesh boundary edge segments (straight lines) diat are die 

input to the constrained Delaunay ttiangulation. 
20 The mesh is finally obtained by applying constrained Delaunay 

triangulation to die set of decoded node points, where die edge segments on die mesh 
boundary are used as constraints. Delaunay diangulation proceeds by inserting an edge 
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beweea two node points if there exists a circle through these two points that does not 
contain any other node point in its interior and does not intersect any other node poinL 
The boundary edge segments present the only exceptions to this definition and may not 
be removed in the process. An example of a mesh obtained by constrained triangulation 
5 of the node points of Fig. 11 and is shown in Fig. 12. 
Dprnriino of T^Qdft Motion Vectors 

Each node point p. of a 2D mesh numbered t in the sequence of meshes 

has a 2D motion vector v. . defined from mesh k to ^+1. By decoding aU motion vectors 

n _ 0 iV - 1, one is able to reconstrua the locations of node points p; of the mesh 

10 numbered ifc+1. Note that the triangular topology of the mesh remains the same 
throughout the sequence of meshes. 

As previously explained in the section entitled "Encoding of Node Motion 
Vectors", two mediods for motion coding are described. Accordingly, two methods for 
motion vector decoding will be described and are identified as Method I and Method E. 
15 Method I 

Method I is based on a predictivie coding technique where only one 
motion vector is used as a predictor. A unique ordering of the node points p. is 
available, where n denotes the ordering number. In the case that the mesh represents a 
Delaunay-type mesh, this ordering is simply defined as the order in which the node point 
20 locations are received during mesh geomeny decoding, "nie node point for which the 
location was received first is Po . the node point for which die location was received after 
that is p,. etc. In the case where the mesh is a uniform-type mesh, the node ordering is 
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on basis of .h. m«h as previously described m *= section endUed 

encoding ot Node Motion Vectors: Method I." TOs orterifg. defined by a simple 
oavetsal of die luiif onn mesh, is intETOled in H|. «. 

For each node poift in the mesh, a one bit Dag specifies whether that node 
point has a non-zero motion vector. For each node point in d« mesh with non-zeto 
motion vector, a modon vector *ff.re„c. vec. e. specifies the difference between me 
motion vector of tfiat node and the previously decoded modon vecmr. Tie decoding 
process is as foUows: To decode ^ motion vectors, vectors ^ =re retrieved from the bit 
stream one by one. and used to compute the new locations of node points in the order 
specified above. If d,e first node point in U>e ordering has a zero motion vector, then: 

5. =(0.0) . ('> 
ff the first node point in the ordering has a noo-zero motion vector, then 
rhe firs, vector retrieved from the bit stream. ^. and specifies *e motion vector of the 
grst node point directly: 

' - r (10) 

Vo = Co • 

The new location of the fiist node point is obtained by: 

All other vectors retrieved from the bit stream e. are motion vector difference vectors 
and are added to the previously decoded motion vector to obtain the actual motion 

vector. If node point p. . n = 1 l^- h bas a zero motion vector, then: 

v=(0.0). (12) 

otherwise: 
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The new location of node points p. . n = 1 - 1. is obtained by: 

p-' = P. + v.. (U) 

l^prhod n: M 9""n Vector Prediction 

Method n is based on a predictive coding technique where two motion 
vectors are used to compute a predictor. The predictive decoding technique of Method E 
specificaUy employs the foUowing priiiciple: To decode the motion vector of a node 
point p. thai is part of a triangle t, = (p,. p„. p.), where the two motion vectors vectors 
V. and V of the nodes p, and p„ have already been decoded, the values of v, and v„ 
may be used to predict v. and decode the prediction error value. Starting from an initial 
triangle t^. of which afl three node motion vectors have been decoded, there must be at 
least one other, neighboring, triangle t. that has two nodes in common with t,. Because 
the motion vectors of the two nodes that t, and t, have in common have already been 
decoded, these two motion vectors may be used to predict the modon vector of the third 
node in t,. The acmal prediction vector w. is computed by averaging of the two 
prediction morion vectors and the components of the prediction vector are roundedto 
half-pixel accuracy, as follows 

w =03x(|.v, + ^^+0JJ). (15) 

The motion vector itself is computed by: 



v, = w,-!-e. . 



(16) 
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Where denotes tbe prediction error vector, the components of which are decoded from 
variable length codes. TTiis procedure is repeated while traversing the triangles and 
nodes of the mesh, as explained below. Whfle visiting all triangles of the mesh, the 
motion vector data of each node is decoded from the bit stream one by one. Note that no 
prediction is used to decode the first motion vector. 

and that only the first encoded motion veaor is used as a predictor to encode the second 
motion vector, 

V =v (18) 
It Should be noted diat the prediction error vector is specified only for 
node points with a non-zero motion vector. For all other node points, the motion vector 
is simply v„ = (0.0). 

MPthofi TT: M ftri^ Traversal 

A breadth-first Traversal is used to visit all the triangles and nodes in the 

15 mesh nmnbered fc. and to decode the motion vectors defined from mesh * to i+1. The 
breadth-first traversal of the triangles is defined in the section on "Encoding of Node 
Motion Vectors: Method n." anil may be applied simHariy to the decoding phase. 

The ordering of the triangles according to their label numbers assigned 
during the breadth-first traversal impficitly defines the order in which the motion vector 

20 data of each node point is decoded, as described in die foUowing. Initially, motion 
vectordaiafordie top-left nodeofthemeshisretrievedfrom the bitstream- No 

prediction is used for die motion vector of this node, hence diis data specifies die motion 
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vector itself. Motion vector data for the second node, which is the next clockwise node 
on the boundary with respect to the top-left node, is retrieved from the bit stream. This 
data contains the prediction error for the motion vector of this node, where the motion 
vector of the top-left node is used as a prediction. These first two nodes (that form the 
base edge of the initial triangle) are marked with the label 'done'. 

During each iteration of the breadth-first traversal as described before and 
starting from the initial triangle^ the motion vectors of the two nodes that are on the base 
edge of the current triangle during that iteration are used to form a prediction for the 
motion vector of the third node of diat triangle. If that third node is not yet labeled 
'done', morion vector data is retrieved from the bit stream and used as prediction error 
values, Le., the decoded values are added to the prediction to obtain the acmal motion 
vector. That third node is then labeled 'done'. If the third note is already labeled 'done', 
then it is §imply ignored and no data is retrieved from the bit stream. Due to the nature 
of the traversal process, the two nodes on the base edge of a triangle are guaranteed to be 
labeled 'done* when rfiat triangle becomes the 'current triangle\ signifying that their 
motion vectors have already been decoded and may be used as predictors. 

Because every niangie in the mesh has at least one adjacent neighbor, and 
triangles are labeled when visited, every niangle is visited exactiy once and the traversal 
terminates when (and only when) all triangles have been visited. Node points may be 
visited more than once, but their motion vectors are decoded only at the first time a node 
is visited and each node is labeled at the time of decoding, therefore, no attempt will be 
made to decode the corresponding motion vector again. The unique ordering of node 
points corresponds to the order of motion vector daia present in the bit stream. 
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SimUar to the encoding phase, the breadth-first traversal of the triangles 
and the decoding of the node motion vector data may be performed simultaneously by 
maJdng use of a first-in-first-out (FIFO) queue of triangles. For details, see the section 
on "Encoding of Node Motion Vectors," above, and Fig. 10. which illustrate the breadth- 
first traversal process during decoding for a small triangular mesh. Hg. 10 depicts a 

traversal of triangles arbitrarily numbered t, t, and the corresponding ordering of . 

node points arbitrarily numbered p„ p,. The mitial triangle, t,. is defined using the 

top-leftnode p, and ±e next clockwise node on die boundary p. The motion vector v, 
is decoded without using any prediction. Tht motion vector v, is decoded using only v, 
as a prediction. Nodes p, and p, are marked 'done'. Tlie triangle queue is initialized 

with 13, and t, is then labeled. 

The actual breadth-first traversal of the mesh starts by removing the front 
element from the queue, in this case - The motion vector of its third node, v, . is now 
decoded using ^ and v, as predictors and retrieving the prediction error from the bit 
stream. Node p, is mariced 'done'. The next step is to append any triangles adjacent to 
the current triangle t, that are not yet labeled to die triangle queue. The triangle on the 
right of the current triangle Cm this case t, ) is appended first; then the triangle on ±e left 
(in this case to); both aie labeled. The next iteration of the traversal commences by 
removing the frontelementfrom the queue, in this case t,. The motion vector of its third 
node. >^ . is now decoded using -^ and v. as pre^Uciors and retrieving &e prediction error 
from the bit stream. Node p, is mari^d 'done' and the comiected triangle t, is appended 
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to the triangle queue and labeled. Such iterations continue until the triangle queue is 
empty and all node motion vectors have been decoded; the steps of the algorithm are also 
illustrated in Table 2. 
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triangle queue state 
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Table 2 



The Bit Stream Syntax and Semantics 

The following is an implementation of the inventioa within the current 
version of MPEG-4 Worldng Draft (WD )V 5.0. The Working Draft specifies only the 
10 decoding process. The following is an object-oriented pseudo computer code 

implementing the decoding process for both the mesh geometry and motion vectors 
according to the structure of the current specification of MPEG-4 WD V 5.0. 
Mesh Object 

The Me^ Object class defines the syntax for a sequence of 2D meshes. 
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MeshObjectO ( 


No. of bits 


Mnemoxuc 


mesh object start code 


32 


DSiuI 


do { 






MesbObjectPlaneO 






} while (oexibits.bytEalignedO = 
mesfa_object_ttoie_garT_code) ^ 






next start.codeO 






aicsh_pbjcct_eitd_code 


32 


bslbf 


} 







Tables 



M'"^h nh^Rct Plane 

The Mesh Object Plane class defines the syntax for coding a single 2D 

mesh, either as a new mesh or with respect to the previously coded mesh. 



MeshObjeciPlaneO { " 


No* of bits 


Mnemonic 


mesh_object_plane^start_code ^ 


32 


bslbf 


new mesh_flag — : 


1 


bsblf 


if (new mesh flag= *!') { 






MeshGeomaiyO 






} ■ 






else { — ^ — — 






MeshModonO 






} - , 






} : - — 







Table 4 

10 The Mesh Geometry class defines the syntax for coding the 2D geometry of a new 
mesh, which may either be a nnifonn mesh or a Delaunay triangulated mesh. 
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MeshGeometryO { 


No. of bits 


Mnemonic 


m€sh_type ^code 


2 


bslbf 


if (mesh type code= W) { 






nr_of_roesh„iiodes_hori ^ 


10 


uimsbf 


nr of mesh nodes verti 


10 


uimsbf 


mesh_rect_size_hori 


8 


uimsbf 


mesh reel size^verti 


8 


mmsbf 


triancle split code 


2 


bslbf 


} , ^ 






else if (mesh,cype_code = X)l ') { ^ 






nr_of_mc^_nod€S . 


lu 




nr^oCboandaryjodes 


1ft 




aodeO_x 


1ft 




nodeO^y 


1ft 




far (n-1; a < nr of_mesh nodes; d-m-) { 






delta X len v|c 


0 O 


vldbf 


delta X 


\j~Lx 


vldbf 


delta y— teii_yic 


2-9 


vldbf 


delta^y 


0-11 


vlcttrf 


} 






} 






} 







Tables 



The Mesh Motion class defines the syntax for coding the 2D motion vectors of the previous 
mesh to die current mesh, thereby coding the current 2D mesh. 
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10 



Me shMoaonQ ( ^ 

tpotio n^range.code 

for ( D=0; D < PT.of.mesb.Dodes; imh-) { 
Dode_ rootion_vector,Oag 
if (node. modon-vector.flag = '1') { 



delta_invjc_res 
delta_inv_y_vlc 



delta_inv_y__res 



1-13 



0- 2 

1- 13 



0-2 



Yidbf 



vidbf 



uimsbf 



Tables 

fi^miinTiC^ Syntax 

The mesh bit stream syntax consists of two pans: mesh geometry and 

mesh motion. The mesh geometry is only encoded, when a new mesh needs to be 
i^alized; it consists of d.e initial positions of the mesh nodes. The mesh motion is 
encoded at subsequent time instants to describe the motion of the dynamic mesh; it 
consists of a motion vector for each mesh node. 

The dynamic mesh syntax allows for coding . different types of 2D mesh 
strucmres; e.g., uniform or Delatmay-constrained and differem magnitude ranges for the 
node motion vector. The node coordinate and node motion vector accuracy is 0.5 pixeL 
^er.bit - one-bit code is set to "1". Setting this bit to "1" prevents emulation 

of Stan codes. 
15 mesh.object.start_code 

Tins is the bit string 'OOOOOIBC' in hexadecimal. It marics the start of a mesh object. 
This is a unique bit string of length 32 bits for the purpose of synchronization. 
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mesh_object_end_cod€ 

This is the bit string in hexadecimal. It marks the end of a mesh objecL This is a unique 
code of length 32 bits for die purpose of synchronization. 
inesh_object_pIane_start_code 
5 This is die bit string *(X)0001BD' in hexadecimal. It marks die start of a mesh object 
plane- This is a unique code of length 32 bits for die purpose of synchronization- 
new_mesh_flag 

This is a one bit code specifying whedier a new mesh is to be initialized or that the 
current mesh is encoded widi respect to die previous mesh. 



10 



15 



Bevr_nicsh_flag 


mesh coding type 


1 
0 


I (coded by geometry) 

P fcoded bv modOB with resoect to previous mesh 



Table? 
mesh_type_code 

This is a one bit code specifying die type of initial mesh geometry being encoded. 



mesh tvpe code 


mesh geometry tvpe 


00 


noifonn 


01 


Deiamiay 


10 


not used (reserved) 


11 


mt used (reserved) 


Tables 





iir_of_mesh_nodes_hori 

Tlus is a 10 bit code specifying die ntimber of nodes in one row of a oniform mesh. 
nr_of_mesh_nodes_verti - 

This is a 10 bit code specifying die number of nodes in one column of a uniform mesh. 
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mesh_rec±_si2e_hori 

T^Ua8bUcoaespeci£y^g*e-<'thofa«-Sleof..mifono«esh(~g 

two triangles) in half pixel units. 
ine*_rect_size_ver1i 

™.ila8bitco<tespecifyinga.ek=igh.of.,.c.^gleofau»fonnmesh(co.^ 

two triangles) in half pixel units- 
trian^e_split_code 

TWsUa2bi. code specifying bow ^ctaagfcsof.umfonu mesh are sputtojom 

triangles. 




00 



01 
10 

11 



spin au ic*,*<i"5**-» lop*lcft ' — 

cnlif an lectan^ bottom-left to top-nglii 



Table 9 



nr_of_inesh_nodes 

J a 16 bi. code aeauBg u,^ n.mb« of node, (vetuces) of a (non-unifoon) 
Delauna, mes>. These .odes taclude bo* intenor nodes as weU as boundary nodes. 
nr_of_boaBdary_nodes 

W J a 10 bi: code deftomg d^ n-ber of .odes (verdces) on ^ boundary of a (non- 

imif onn) Delaunay mesh. 

Tl^ is a 10 bit code specifying the x^oonlinate of *e first boondaty 
mesh in half-pixel uniis with respea to a load coordinate system. 
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nodeO_y 

This is a 10 bit code specifying the y-coordinate of the first boundary node (vertex) of a 

mesh in half-pixel units with respect to a local coordinate system. 

ddta_xjen_vlc 

5 This is a variable length code specifying the length of the delta.x code that follows. The 
delta.x Jen.vlc and dclta_x codes together specify the difference between the x- 
coordinates of a node (vertex) and the previously encoded node (vertex). The definition 
of die delta_x_lcn_vic and delta_x codes are given in the MPEG-4 Video Verification 
Model 7.0 (VM 7.0) (N1642. section 4.7.1.1) (April 1997), hereinafter MPEG-4 WM 

10 7.0. ^^*le^e they are used for sprite ttajectory coding. 
delta_x 

Tiis-is a fixed length code defining the value of the difference between die x-coordinates 
of a node (vertex) and die previously encoded node (vertex). The definition of die 
- delta_xjen_vlc and delta_x codes are given in the MPEG-4 WM 7.0, section 4.7.1.1, 
15 where tiiey are used for sprite trajectory coding. 
delta_j_Ien_vlc 

This is a variable lengdi code specifying die lengdi of die delta_y code diat foUows. THe 
delta_jjen_vlc and ddta_y codes togedier specify die difference between die y- 
coordinates of a node (vertex) and die previously encoded node (vertex). The definition 
20 of die delta_y_len_ylc and deltas codes are given in die MPEG-4 WM 7.0. section 
4.7.1.1, where diey are used for sprite trajectory coding. 



37 



PCr/JP97/04d07 

WO 98/27515 



dc!ta_y 



ae.ta^j™.,.c aB^d-a^ co^esa. s^-in .e MPECM WM 7.0. s=«ioo4.7.M. 
inotion_range_code 

THs a 2 W. code ^=01,^8 ^ »' """^ '"^^ 



inotion_range_code 


motion vector range in 
half sample units 


1 
2 

^ 


(-3Z311 
[-64,631 



Table 10 
node_motion_vector_flag 

• 1 H> «>de soedfyina whether a motion veaor follows. If a node has a zero 

10 This is a 1 bit code specuym:, wu 

„„d» vector.to*emodoo veaor is »o: encode* to^aia 
,0<....-o,.™c«..0a...v.ue^.^. .d.n.U3e.u>"r.Un.ea.^ 
node has a non-zero motion vector. 
deita_niv_x_vlc 

, ^.^ava.^ie.e.^c.^-^i'-^'--"*''"-""-^-"''"'^"*'''''" 
afflerenceiad.e..c»po«ntofU,en.o^o..ec»ofanodec„n.pared..hex- 

c„.,™e„.of apredi^S-^i— ^- 3ec.,« o„ *e decodes P-3S.<^- 
howtbeptediaingmooottveciorisfomd. ineoe 

de.U.n,..x.res codes are pve. ia d. MPEO-4 WM 7.0. secdoa 6.1.8 and 6. 1.9. 
.0 Table30,whe«d.,a«-«'t«--"««-'««"="""'*'- 
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d€lta_mv_x_res 

This is a fixed length code defining the residual in die difference of motion vectors as 
encoded by d€lta_inv_x_vlc. The definition of the delta_mv_x_vlc and 
delta_mv_x_res codes are given in die MPEG-4 WM 7.0. section 6.1.8 and 6.1.9, 
5 where diey are used for motion vector difference coding. 
delta_inv_y_Tic 

This is a variable length code defining (together widi delta_my_y_res) die valne of the 
difference in die y-component of die motion vector of a node compared to die y- 
component of a predicting motion vector. The section on die decoding process describes 
10 how die predicting motion vector is found. Tne definition of die delta_mv_y_vlc and 
delta_mv_y_res codes are given in die MPEG-4 WM 7.0 section 6.1.8 and 6.1.9. Table 
30. where diey are used for morion vector difference coding. 
delta_mv_y_res 

This is a fixed lengtii code defining die residual in die difference of morion vectors as 
15 encoded by delta_mv_y_vlc The definition of ■ the deIta_inT_y_vlc and 
delta_mv_y_res codes are given in die MPEG-4 WM 7.0 secrion 6.1.8 and 6.1.9. where 

diey are used for motion vector 'difference coding. 

Tn^^sTriai Applications 
The 2D dynamic mesh representation supports die following applications: 

20 Video Obi^ ^Tanipnlarion 

In 3D computer graphics, die animation parameters of a 3D mesh model 
are often syndieticaUy specified. The animation parameiers of a 2D mesh model may 
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B^wise be specific sy..he«c^y, also be denved ftom a -anTBl video 
^uence b, motion esdxnadoo. Tios, mesh modeUng alows us u, iBcen^veb^ combine 

.„d s,..heuc obiec wi*in a .dfied ftamewott Possible appUcanoBS incl^ie 
Object .lansnguratton (replacing one object by ano*er object in moving video). 
3 augmented reaHty (augmenting/overlaying an object in moving video witb graphics or 
^t) and spado/temporal image interpolatio. (zooming or frame-rate converdon) 

MoUon estimalion and compensation have proven to be impdnant tools . 
forvideocodingsystemstoobtainhighcompres^onranos. Theproperdesofd^mesh- 

,0 based moUon model mate u a suitable alternative to bloclc-based motion modeling for 
^asamodoncompensationtoo, in videocoding in ^hichca^ the modon vectors of 

^oodesarenansminedinsteadofbioct-modonvectors. Mesh-based video 
compress^nmay tor instance beperformedby^tdngte^ne maps only at selected 

toy frames and by animating these te^re maps (widtou. sending any predicdcn error 
^ taage)fortheintennedia.efiamesns.ng2Dmeshinfonnation. Only the in^ge data o, 
:,.fi:.tfiameofase<prenceise.coded.asweuas,heme^modoudataconespondmg 

to all ftames of the setpence. In ease dre iuidal mesh is adapted to image contenu 
infomtarionahoutd^imtialme* geometry has .0 be encoded andtransroitted as welL 

I„ d« case of video coding, the inidal mesh geometry is sometin.es restricts 

» ovedtead mvolv^l (e-g.. nmform meshes,: however, more general mesh geometry 
compress.onschemesm,ybeapplied,ncaseofmulnmediacoding.«heresev=«l 

objecrs (video, audio, text. 21. / 3D meshes etc.) are encdded independendy. 
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Jndfr i^inp and Brpw.sing for Video LiT^raries 

Content-based random access to individual video objects becomes a 
desirable feature in several distributed multimedia appUcations. such as object-based 
browsing/editing/manipulation of video databases. Useful motion and shape features 
5 may be derived from the 2D content-based triangular mesh representation of video which 
may be employed in such applications. 

Thus, a system and method for a 2-D mesh geometry and motion vector 
compression tove been disclosed. Although a preferred embodiment of the method of 
the invention, and several variations thereof, have been disclosed, it should be 
10 appreciated that further variations and modifications may be made thereto without 
departing from the scope of the invention as defined in the appended claims. 
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CLAIMS 

1 A method of encoding and decoding a dynamic mesh, comprising: 

encoding and decoding a mesh geometry- of a set of node points; and 
encoding and decoding a mesh node motion vector for each node point. 

2. ' The method of claim 1 wherein said encoding and decoding of a mesh 
geometryincludescompressingameshtopologyofaDelam.aytype. 

3. The method of claim 1 wherein said encoding and decoding of a mesh 
- includes compressing a mesh topology of a uniform type. 



geometry i 



The method of claim 1 wherein said encoding and decoding of a mesh 



sing the motion vectors for each node point using a linear 



4. 

node modon vector includes: 
compressing 1 

ttaversal of the node points and 

predicting, using first order prediction, subsequent motion veaors for 

each node poiai. 

5. The method of claim 4 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes. 

6. - Themethodofclaim4whichincludestraver^gandorderingthemesh 
node points for a Delaunay type mesh. 
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7 The method of claim 4 which includes traversiQg and ordering the mesh 
node points for a uniform type mesh. 

8 The method of claim 1 wherein said encoding and decoding of a mesh. 

node motion vector includes: 

compressing the motion vectors for each node point using a breadth-first 

traversal of the node points; and 

predicting, using second order prediction, subsequent motion vectors for 

each node point 

9^ The method of claim 8 wherein said predicting includes encoding and 

decoding prediction errors using variable length codes. 

10. A method of encoding and decoding a dynamic mesh haviug a set of 
polygonal patches therein, comprising: 

encoding and decoding a mesh geometry of a set of vertices; and 
encoding and decoding a mesh node motion vector for each vertex. 

1 1. The method of claim 10 wherein said encoding and decoding of a mesh 
geometry includes compressiog a mesh topology of a Delaunay type. 

12. The method of claim 10 wherein said encoding and decoding of a mesh 
geometry includes compressing a mesh topology of a uniform type. 
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13. The method of claim 10 wherein the polygons are triangles and said 

encoding and decoding of a mesh node motion vector includes: 

compressing the motion vectors for each vertex using a linear rraversal of 

the vertices; and 

predicting, using first order prediction, subsequent modon vectors for 

- each vertex. 

14. THe method .of claim 13 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes. 

15. Tl.e method of claim 13 which includes traversing and ordering the mesh 
vertices for a Delaonay type mesh. 

16. The method of daim 13 which includes traversing and ordering the mesh 
vertices for a uniform type mesh. 

17. The method of claim 10 wherein said encoding and decoding of a mesh 

node modon vector includes: 

defining triangles throughout the mesh; 

compressing *e modon vectors for each vertex using a breadth-first 

traversal of die triagles aiui Ae vertices of die triangles; and 

predicting, using second order prediction, subsequent motion vectors for 

each vertex. 
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18. The method of claim 17 wherein said predicting includes encoding and 
decoding prediction errors using variable length codes. 

19. A system for encoding and decoding an image represented by a dynamic 
mesh, wherein the dynamic mesh includes a predefined geometry, having a set of node 
points and motion vectors for the node points; comprising: 

an encoder subsystem for encoding the mesh geometry of the set of node 
points and encoding a mesh node motion vector for each node point including: 

a texture encoder for encoding the texture of an image and for 

producing an image biistream therefrom; 

a mesh encoder for encoding a mesh sequence; said mesh encoder 
including a geometry encoder for encoding the geomeuy of the mesh, and a motion 
vector encoder for encoding the motion vectors of the mesh; and for generating a mesh 

coded bitstream therefrom; 

a decoder subsyrstem for decoding the mesh geometry of the set of 
node points and decoding a mesh node motion vector for each node point including: 

a texture decoder for decoding die texture of an image from an 

image bitstream; 

a mesh decoder for decoding said mesh encoded bitstream, 
including a geometry decoder for decoding die geometry of the mesh, and a motion 
vector decoder for decoding die motion vectors of die mesh; and 

an animation application for combining die decoded texmre data and the 
decoded mesh, and for generating an animated image therefrom. 
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